Materials for Domain-general modal cognition
Eli Hecht & Jonathan Phillips
March 2024
Summary
This file contains the materials for the paper “Domain-general modal cognition” by Eli Hecht and Jonathan Phillips. Here you will find information on all study procedures and analysis that are reported in the main text and supplementary information.
Study 1: Possibility Generation
Participants
Across three studies (\(N_{1}\) = 104, \(N_{2}\) = 104, \(N_{3}\) = 103) we collected a total sample of 311 participants (\(M_{age}\) = 37.36; \(SD_{age}\) = 10.71; 131 females) from Amazon Mechanical Turk (www.mturk.com).
Procedure
In each study, participants were presented with six stories, what we’re referring to as ‘decision-making contexts’, in random order, each describing an agent having to make a decision. All 18 contexts are presented in the ‘text’ column of Table 1.
For each story, participants were asked:
After listing their responses for each scenario, participants were then asked to rate each of their answers on three 100 point scales.
Results
10 participants gave nonsensical answers and their responses were excluded from further analysis.
Participants’ first responses for each scenario tended to be the one they rated highest, or tied for highest (40.9% of the time).
The better an option is, the earlier it tended to be generated by a participant (\(F\)(1\() =\) 638.25, \(p <\) 0.001). Further, independent of normality and morality, the more probable a response was, the more likely it was to be generated earlier (\(\chi^2\)(1) = 57.01, \(p <\) 0.001). The same is true for morality, where independent of the other ratings, the more moral a possibility was, the more likely it was to be generated earlier (\(\chi^2\)(1) = 25.72, \(p <\) 0.001). Normality was not found to have a similar significant independent predictive effect, (\(\chi^2\)(1) = 2.731, \(p =\) 0.098). This is likely because, as will be seen below, normality is jointly predicted by probability and morality, much of the variance in answer generation number is already accounted for by these ratings.
We sought to replicate the finding of Bear and Knobe (2017) that normality judgments are predicted by judgments of morality and probability. First we constructed a linear mixed-effects model to predict normality ratings with probability and morality ratings as predictors and fixed effects for context and [not positive on how to refer to the structure of the model]. We further found that both ratings were independently predictive of normality. The model performed worse when probability was removed (\(\chi^2\)(1) = 480.159, \(p <\) 0.001), and when morality was removed (\(\chi^2\)(1) = 293.965, \(p <\) 0.001). Importantly, the model performed significantly worse when the interaction between morality and probability were removed (\(\chi^2\)(1) = 18.56, \(p <\) 0.001).
Manually grouping participant responses
Two raters manually coded each of the 9150 participant responses into 13-18 distinct action categories for each context. The criterion for a grouping was that at least three participants generated an action within that category. Two raters independently grouped participant responses with an inter-rater agreement of 82.3% and a Cohen’s kappa of 0.806 indicating strong inter-rater reliability. A third rater determined final results in cases of disagreements. Action categories for each context are presented in Table 2.
Results
We found striking convergence across participant’s answers within
each context. Only 5% of answers were labeled as ‘other.’ Across
contexts, only 3% of participants did not come up with any of the 3 most
common answers in their set of 5 possible actions, and only 20% of
participants didn’t put down the most popular answer for that
context.
Not only did participants converge on a relatively small set of action
categories (13-18), but the most common answers also tended to be rated
highly. Across contexts, the actions in the three most common categories
were rated higher (\(M =\) 74.7, \(SD =\) 25.2) than other actions (\(M =\) 57.4, \(SD
=\) 32.5).
This pattern held true within each context. For each context, we
performed an independent samples t-test comparing average ratings for
answers in the three most popular action categories to answers outside
of these categories. For all but 3 of the 18 contexts t-values were in
the expected direction, with \(p <
.001\).
Algorithmic grouping of participant responses
In addition to manually coding participant responses, we also used natural language processing to group similar responses. We computed sentence embeddings for all responses in each scenario using Sentence-BERT and reduced the dimensionality of the embedding space using UMAP. We clustered the resulting embeddings using the DBSCAN algorithm. We selected values for \(\epsilon\), or neighborhood size, and minimum number of samples by testing a range of values and visually inspecting the resulting the plots. The analysis used is provided in nlpCoding.ipynb.
We selected values of \(\epsilon =\) 0.4 and \(minSamples =\) 6 as the parameters that resulted in the most semantically-meaningful clusters across contexts. At these values, the number of clusters per context ranged from 11 to 20.
Results
Using the clustering results with the DBSCAN alogirthm, as with manually grouped responses, we found striking convergence across participant’s answers within each context. Only 11.6% of answers were labeled as ‘outliers’ by the algorithm, analogous to our finding for ‘other’ responses using manual coding. Across contexts, only 7.2% of participants did not come up with any of the 3 most common answers in their set of 5 possible actions, and only 33% of participants didn’t put down the most popular answer for that context.
Unlike with the manual coding results, the actions in the three most common DBSCAN clusters were not significantly higher (\(M =\) 67.4, \(SD =\) 30.3) than other actions (\(M =\) 63.4, \(SD =\) 30.9).
Study 2: First-person decision making
Participants
We collected a sample of 104 participants (\(M_{age}\) = 38.89; \(SD_{age}\) = 11.05; 50 females) from Amazon Mechanical Turk (www.mturk.com). 4 participants did not finish the study and we’re excluded from further analysis. 7 participants gave nonsensical/unclear responses, so their data was also excluded from analysis.
Procedure
We modified our original 18 scenarios, replacing all names and pronouns referring to the agent with second-person pronouns (see Table 1 for the modified scenarios). Each participant viewed all 18 contexts with modified pronouns in randomized order. After reading each context, participants were asked “In this situation, what would you decide to do?” and were given a free-response text box to write their answer within.
Manual grouping
Once again, two raters manually coded each of the 1800 participant responses into 13-18 distinct action categories for each context. The same action categories as were found for Study 1 previously were used (see Table 2). Two raters independently grouped participant responses with an inter-rater agreement of 82.6% and a Cohen’s kappa of 0.799 indicating strong inter-rater reliability. A third and a fourth rater determined final results in cases of disagreements.
Results
We constructed a model to predict the likelihood of a person selecting a response in a given cluster as their decision. In-keeping with pre-existing two-step models of decision-making (Morris et al.), for each context, the algorithm first sampled a consideration set of participant responses from Study 1. It assigned a value to each response based on the average participant ratings of responses from the cluster it was assigned and selected the response within each set with the highest value. We then calculated the likelihood of answers from each cluster being selected by this model across 10^{4} sampled consideration sets.
We ran this analysis with consideration set size varying from 1-10. At consideration size \(k = 1\), the model proportion for a cluster is equal to the likelihood of initially sampling that cluster. At all other consideration set sizes, the average value of the cluster also plays a role. As consideration size increases, initial sampling likelihood plays a less significant role and average value is more significant, with low-probability but high-value actions more likely to be selected by the model.
We then calculated the correlation between the likelihood of a
response from a given cluster being selected by this model and the
likelihood of a participant selecting a response from the same cluster
as their decision. The correlation was significant at all consideration
set sizes tested (0.637 \(<r<\)
0.865, all at \(p< 0.001\)). It was
highest at a consideration set size of \(k
=\) 2. At consideration set size \(k=3\) the decision model had a correlation
of \(r=\)
round(k_correlations$correlation[3],3).
We also re-ran the model three times, separately using average probability, average morality, and average normality of responses for each cluster as our measure of cluster value instead of using the average of these three ratings. Using average probability, the model performed similarly using all three ratings (0.636 \(<r<\) 0.863 using probability ratings, 0.637 \(<r<\) 0.867 using morality ratings, and 0.638 \(<r<\) 0.865 using normality ratings, all at \(p<.001\)). At consideration set size \(k=3\) the decision model using probability ratings had a correlation of \(r=\) 0.838, the decision model using morality ratings had a correlation of \(r=\) 0.837, and the decision model using normality ratings had a correlation of \(r=\) 0.837.
Algorithmic grouping of participant responses
Once again, in addition to manually coding participant responses, we also used natural language processing to group similar responses. For decision analysis, we aggregated the possibility generation from Study 1 and the decisions from Study 2. This allowed us to have clusters with identifiable values, using the ratings from the first study, which we could then use to predict the likelihood of the cluster being chosen as a decision. We computed sentence embeddings for all responses in each scenario using Sentence-BERT and reduced the dimensionality of the embedding space using UMAP. We clustered the resulting embeddings using DBSCAN. We once again used values of \(\epsilon =\) 0.4 and \(minSamples =\) 6 as the parameters that resulted in the most semantically-meaningful clusters across contexts. At these values, the number of clusters per context ranged from 8 to 22.
Results
We used the same decision making model as previously, this time predicting the likelihood of a decision in a given DBSCAN cluster rather than a manually coded action category being selected for the first-person decision study.
The model once again strongly predicted actual decision likelihood at all consideration set sizes tested (0.509 \(<r<\) 0.835 using DBSCAN clusters, all \(p\)’s\(<.001\)).
Study 3: Action Ratings
For each context, we created a list of six actions the agent could possibly do. These actions were selected to vary widely along the space of options that come to mind. The six actions for each context are presented in Table 1.
Participants
To verify that the actions we came up with did in fact vary as intended, we collected a sample of 322 participants (\(M_{age}\) = 39.76; \(SD_{age}\) = 11.64; 145 females) from Amazon Mechanical Turk (www.mturk.com) to rate the actions.
Procedure
Each participant viewed a random subset of six of the eighteen contexts. For each context, in addition to reading the story, they were randomly assigned one of the six actions and told that the agent was considering that action. They were then asked to rate that action on the same three 100-point scales used earlier, measuring the probability, morality and normality of the action.
Results
The ‘actual actions’ we generated varied widely across all contexts. We averaged participant responses for each action as our measure of action value for each dimension. The average probability ratings of the actions ranged from 1.05 to 92.17, \(M =\) 45.58, \(SD =\) 28.61, the average morality ratings ranged from 1.22 to 97.5, \(M =\) 52.44, \(SD =\) 34.81, and the average normality ratings ranged from 0.68 to 93.83, \(M =\) 45.81, \(SD =\) 32.48.
We averaged all three ratings for each action and took this average as a measure of an action’s ‘value.’
Calculating the modal distance
In order to predict participants’ higher level judgments for these generated actions, we developed a novel measure that compared ratings for each action against the set of all participant generated responses for the context. This value, what we’re referring to as the modal distance, is the proportion of participant generated responses for the context (from Study 1) that had a higher average rating than the average rating for a given action (from Study 2). The less normal an answer is, the higher its modal distance value will be (i.e. the more distant it is from the center of the contextual modal space). Importantly, the value is context dependent: an action with an average normality rating of 50 will have a lower modal distance in a context with skewed positive responses than in a context with a more uniform distribution. This value will be used to predict responses in all subsequent studies. We also sought to see whether each individual judgment (probability, morality and normality) was separately predictive of high-level judgments. We recalculated the modal distance for each action, except instead of using the averages of probability, morality and normality as the value score for each participant-generated and actual action, we calculated three different scores, one for each type of judgment. This gave use a probability distance, a morality distance and normality distance for each action.
Study 4: Force judgments
Participants
We collected a sample of 322 participants (\(M_{age}\) = 39.74; \(SD_{age}\) = 12.53; 185 females) from Amazon Mechanical Turk (www.mturk.com).
Procedure
Each participant was randomly presented with 12 of the 18 contexts. For each context, participants were randomly assigned one of the six actions and told that the agent decided to do that action. They then rated their agreement with a statement that the agent was forced to complete the action on a 100 point scale.
Results
Due to a typo in the question for context 11 (the question referred to a different agent than the main context text), results for this context were excluded from analysis.
Across all actions, participants reported a wide range of agreement with statements of force attribution (\(M =\) 40.9, \(SD =\) 39.0). As expected, using the modal distance value to predict average force judgments for each action, we found a correlation of \(r =\) -0.896, \(p <\) 0.001. The moral distance had a correlation of \(r =\) -0.816, \(p <\) 0.001 with force judgments; probability distance had a correlation of \(r =\) -0.9, \(p <\) 0.001 and normality distance had a correlation of \(r =\) -0.889, \(p <\) 0.001.
Study 5: Causal judgments
For each context, we came up with one downstream consequence that could potentially occur regardless of which of the six actions the agent decided to. Downstream consequences can be found in Table 1. We sought to predict participants’ ratings of whether the action chosen by the agent caused these downstream consequences using our existing modal distance value. These judgments don’t just reflect participant’s judgments of the actual action but also the relationship between the action and downstream consequence. We also sought to compare our model to an existing model of causal judgments that involves judgments of whether a potential cause was sufficient for the outcome. We collected data in three studies, the first on causal judgments, the second on counterfactual relevance and necessity ratings of the action and the third on the sufficiency ratings.
Study 5a: Causal judgments
Participants
For the first study, we collected a sample of 301 participants (\(M_{age}\) = 43.88; \(SD_{age}\) = 13.66; 166 females) from Amazon Mechanical Turk (www.mturk.com).
Procedure
Each participant viewed a randomized subset of 12 of the 18 contexts. For each context, participants read the story and were told that the agent decided to do one of the six actions. They then read that afterwards, another event happened. Participants were then asked to rate their agreement with the following statement on a 100 point scale:
Study 5b: Counterfactual judgments
Participants
For the second study, we collected a sample of 304 participants (\(M_{age}\) = 42.19; \(SD_{age}\) = 12.44; 158 females) from Amazon Mechanical Turk (www.mturk.com).
Procedure
Participants viewed a randomized subset of 12 of the 18 contexts. For each context, participants read the story and were told that the agent decided to do one of the six actions. They then read that afterwards, another event happened. Participants were then asked to rate their agreement with the following statements on 100 point scales:
something other than deciding to [Chosen action].
These statements were modified as necessary to make the sentence flow naturally (e.g. “If Heinz hadn’t decided to [Chosen action], his wife wouldn’t have gotten more ill.”)
Study 5c: Sufficiency judgments
Participants
For a third study, we collected a sample of 274 participants (\(M_{age}\) = 40.89; \(SD_{age}\) = 12.5; 148 females) from Amazon Mechanical Turk (www.mturk.com).
Procedure
Participants viewed a randomized subset of 12 of the 18 contexts. For each context, participants, read the story and were told that the agent decided to do one of the six actions. They then read that afterwards, another event happened. Participants were asked to rate their agreement with the following statement on a 100-point scale:
These statements were modified as necessary to make the sentence flow naturally (e.g. “Given that Heinz decided to [Chosen action], his wife was going to get more ill.”)
Results
Participants reported a wide range of agreement with statements attributing causal responsibility to the agents (\(M=\) 44.2, \(SD=\) 38.5). To predict participants’ judgments of causal atrtribution for each event, we generated a new measure that incorporated the modal distance for each event and participants’ judgments of the counterfactual relevance. The modal distance should only play a role in causal judgments if participants judge the event as necessary for bringing out the downstream consequence. As such, we multiplied participant ratings for counterfactual necessity (data from Study 5b with modal distance values generating a value which we’re calling ‘necessity strength’ for each event. As predicted, across action-context pairs, necessity strength was highly correlated with causal ratings (\(r =\) 0.747, \(p <\) 0.001).
The moral distance had a correlation of \(r =\) 0.547, \(p <\) 0.001 with force judgments; probability distance had a correlation of \(r =\) 0.52, \(p <\) 0.001 and normality distance had a correlation of \(r =\) 0.512, \(p <\) 0.001.
To ensure that both the modal distance and the necessity rating, as well as the interaction between them, were critical for predicting causal ratings, we next conducted a series of model comparisons. Specifically, we built a linear mixed-effects model predicting causal ratings as a function of modal distance, counterfactual ratings and their interaction, with a random effect for context. The full model performed significantly better than a model in which contextual modal distance was removed (\(\chi^2\)(1) = 38.553, \(p <\) 0.001), better than a model in which counterfactual necessity ratings were removed (\(\chi^2\)(1) = 80.359, \(p <\) 0.001), and critically, also better than a model in which the interaction between the two was removed (\(\chi^2\)(1) = 23.551, \(p <\) 0.001). These model comparisons confirm the importance of the relationship between modal distance and necessity: agents are most strongly judged to be causes of outcomes when it is both the case that there were relevant alternative actions that they could have done instead, and if they had done those actions instead, the outcome would likely have been different.
We additionally found evidence that the counterfactual relevance measure used in prior work reflects modal distance: The contextual modal distance of each actual action was highly correlated with participants’ explicit judgments of the relevance of counterfactual alternatives, \(r =\) 0.822, \(p <\) 0.001.
We also constructed a series of linear mixed effects models to compare our model against the existing model of causal judgments. This model includes a term for sufficiency strength, which we calculated by multiplying the probability of sampling an action by participants’ judgments of the sufficiency of the action for causing the downstrema consequence. First we constructed a linear mixed-effects model predicting causal judgments using necessity strength (our measure) and the Icard model’s measure. The model performed significantly worse with our measure removed (\(\chi^2\)(1) = 4.849, \(p <\) 0.028) but not when the Icard measure was removed (\(\chi^2\)(1) = 0.443, \(p =\) 0.506). Because the primary difference between the two measures is that the Icard model includes a term for sufficiency judgments, we constructed another model to see whether sufficiency judgments of the actions contributed anything to causal judgments. We constructed a mixed-effects model predicting causal judgments with necessity strength and sufficiency and a random effect for context. Removing the sufficiency ratings did not significantly effect performance of the model (\(\chi^2\)(1) = 2.521, \(p =\) 0.112).
Study 6: Blame judgments
Using the same stimulus sets as Study 5, we sought to determine whether the modal distance could further be used to predict participants’ moral judgments of the agents. Specifically, we investigated whether participants judged that the agent should be blamed for the downstream consequence occurring.
Participants
We collected a sample of 304 participants (\(M_{age}\) = 41.41; \(SD_{age}\) = 13.89; 175 females) from Amazon Mechanical Turk (www.mturk.com).
Procedure
Study design was identical to Study 5a, except that instead of being asked about causal attribution, they were instead asked to rate their agreement with a statement of blame attribution on a 100-point scale:
Results
Once again, participants reported a wide range of agreement with statements attributing blame to the agent (\(M=\) 40.8, \(SD=\) 37.6).
As with causal judgments, necessity strength was highly correlated with attributions of blame to the agent who acted (\(r =\) 0.732, \(p <\) 0.001). Additionally, we used the same linear mixed effects model we previously used to predict causal judgments to predict attribution of blame. We again selectively lesioned the full model to create three new models: one with modal distance removed, one with counterfactual necessity removed, and one with the interaction term between the two removed. All three models performed significantly worse than the full model (modal distance removed: \(\chi^2\)(1) = 56.118, \(p <\) 0.001; counterfactual necessity removed: \(\chi^2\)(1) = 49.157, \(p <\) 0.001; interaction removed: \(\chi^2\)(1) = 24.209, \(p <\) 0.001). These model comparisons again confirm the importance of the relationship between modal distance and necessity: agents are most held responsible when it is both the case that there were better options available and if they had chosen those alternative options instead, the outcome would have likely been different.
The moral distance had a correlation of \(r =\) 0.637, \(p <\) 0.001 with moral attribution judgments; probability distance had a correlation of \(r =\) 0.595, \(p <\) 0.001 and normality distance had a correlation of \(r =\) 0.599, \(p <\) 0.001.
Appendix
Table 1
| context | text | text_second_person | action_1 | action_2 | action_3 | action_4 | action_5 | action_6 | downstream_event |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Heinz’s wife has recently fallen ill and needs an expensive medication that is not covered by her medical insurance. They don’t have the money needed to purchase the expensive prescription, but they know that it’s vital for her to have it if she is going to recover. | Your spouse has recently fallen ill and needs an expensive medication that is not covered by your medical insurance. You don’t have the money needed to purchase the expensive prescription, but you know that it’s vital for them to have it if they are going to recover. | commit credit card fraud | go on strike for better wages | search for the medication on the black market | start a fundraiser | switch insurance plans | take out a loan | After doing this, Heinz gets into serious financial trouble. His wife has to go without medication, and she soon gets more ill. |
| 2 | Josh is on the way to the airport to catch a flight to a hunting safari in Africa. He leaves with plenty of time to make it there, but his car breaks down on the highway. Now Josh is sitting in his car near a busy intersection, and knows he needs to get to airport soon if he is going to catch his flight. | You are on the way to the airport to catch a flight to a hunting safari in Africa. You leave with plenty of time to make it there, but your car breaks down on the highway. Now you are sitting in your car near a busy intersection, and know you need to get to airport soon if you are going to catch your flight. | call a taxi | demand that the airline delays the flight | reschedule for a later flight | steal a passing driver’s car | take a public bus | try to convince a passing driver to trade cars | Josh does not make his originally scheduled flight, so other people going on the safari have to wait for him. |
| 3 | Brian is the evening manager at a bakery. Near the end of the day, he realizes that $50.75 is missing from the register and that he is responsible for accounting for the money at the end of the day. Brian knows he has to make sure the register is balanced or he might lose his job. | You are the evening manager at a bakery. Near the end of the day, you realize that $50.75 is missing from the register and that you are responsible for accounting for the money at the end of the day. You know you have to make sure the register is balanced or you might lose your job. | explain the situation to his boss | get an advance payday loan | overcharge the next customers | pay the money from his own wallet | sell one of his kidneys | stage a robbery | After the owner of the bakery finds out what happened, she fires the entire staff working that day. |
| 4 | Liz decides to go to the gym on her lunch break to play a game of racquetball with a friend. When she gets to the gym, she realizes that her membership has expired since she last went. Since she left her phone at her desk in the office, she has no way of letting her friend know. | You decide to go to the gym on your lunch break to play a game of racquetball with a friend. When you get to the gym, you realize that your membership has expired since you last went. Since you left your phone at your desk in the office, you have no way of letting your friend know. | borrow someone’s phone | crawl past the front desk | lie that she left her card inside | pay for a day-pass to the gym | pull the fire alarm | renew her membership | Despite doing this, Liz cannot get into the gym, and she misses the racquetball game with her friends. Without Liz, her friends do not have enough players and must cancel the game. |
| 5 | Mary is about to go to her final class of the day when she remembers that there is a homework assignment that is due. Mary’s mom accidentally took her homework assignment out of her backpack when she was making room for her lunch. Now Mary has nothing to turn in for credit. | You are an elementary school student about to go to your final class of the day when you remember that there is a homework assignment that is due. Your mom accidentally took your homework assignment out of your backpack when she was making room for your lunch. Now you have nothing to turn in for credit. | bring in the assignment the next day | call her mom | copy a friend’s assignment | hypnotize the teacher | lie that her house burnt down | quickly try to redo the assignment | After Mary does this, her teacher gives her a zero as her grade for the assignment. |
| 6 | Brad and some friends are hiking through the mountains in the Canadian wilderness. A couple of days into their hike, Brad realizes that they are lost. He knows that a rescue crew could arrive before long, but it is extremely cold and they don’t have much food or water left. | You are hiking with some friends through the mountains in the Canadian wilderness. A couple of days into your hike, you realize that you are lost. You know that a rescue crew could arrive before long, but it is extremely cold and you don’t have much food or water left. | boost morale with a three-legged race | collect wood to start a fire | huddle to conserve energy | search for a water source | steal all of the food and water | throw away all of the food | After Brad does this, he and his friends spend several days lost in the woods, cold, hungry, and thirsty. |
| 7 | Darya is on her way to a concert with her friends. As they approach the entrance her friend Ted realizes he forgot his ticket at his house. The concert is about to start and Ted would likely miss most of the concert if he returned to his house for his ticket. | You are on your way to a concert with your friends. As you approach the entrance your friend Ted realizes he forgot his ticket at his house. The concert is about to start and Ted would likely miss most of the concert if he returns to his house for his ticket. | give her ticket to Ted | go without Ted | pretend they are conjoined twins | steal someone else’s ticket | teleport into the concert | try to buy a ticket off another person | After Darya does this, the concert staff at the entrance become suspicious of Darya and her friends and refuse to let them in. |
| 8 | Eunice is sunbathing at the beach next to a family with young children. When the family is playing in the water, she sees a teenager begin to go through their belongings but she isn’t sure whether or not he’s part of the family. | You are sunbathing at the beach next to a family with young children. When the family is playing in the water, you see a teenager begin to go through their belongings but you’re not sure whether or not he’s part of the family. | ask the parents if they know him | keep an eye on him | make cat noises at him until he leaves | read his mind | tackle him and hold him down | wave to the parents | After Eunice does this, the family begins to feel uncomfortable about Eunice’s behavior and decides to leave the beach for the day. |
| 9 | Eamon is driving home on a two-lane road and thinks the driver ahead of him is intoxicated. The car has been slowly drifting between the two lanes and is currently driving in the middle of road. | You are driving home on a two-lane road and think the driver ahead of you is intoxicated. The car has been slowly drifting between the two lanes and is currently driving in the middle of road. | honk an angry message in Morse code | honk at the driver until they pull over | pull over and call 911 | ram into the car | tailgate the car | take a detour at the next exit | After Eamon does this, he drifting car crashes into another car, resulting in a pileup accident. |
| 10 | Cameron is filling up his car at a gas station on his way home from work when a woman approaches him to ask for a ride to a nearby city. The city is in the same direction as his house but it’s 10 miles further than he planned to go. | You are filling up your car at a gas station on your way home from work when a woman approaches you to ask for a ride to a nearby city. The city is in the same direction as your house but it’s 10 miles further than you planned to go. | call her a cab | refuse to drive her | steal whatever she has on her | take her to his house but no further | turn invisible | yell in gibberish until she leaves | After Cameron tries to do this, the woman attempts to walk all the way to her destination. As a result, the woman is late. |
| 11 | Erica is leaving the mall on a hot summer day. In the parking lot, she notices a dog in the back of a car without any of its windows open. The dog is panting heavily and looks tired. | You are leaving the mall on a hot summer day. In the parking lot, you notice a dog in the back of a car without any of its windows open. The dog is panting heavily and looks tired. | ask to use the mall’s loudspeaker | call emergency services | check that the doors are locked | light a fire under the car | pour water on the car to cool it | slash the tires | The dog dies inside the car before anyone else arrives to help. |
| 12 | Carl is a chef on his way to a food festival in Chicago. He is in line to go through security at the airport when he realizes he accidentally brought a large knife in his bag. | You are a chef on your way to a food festival in Chicago. You are in line to go through security at the airport when you realize you accidentally brought a large knife in your bag. | ask to check the knife | hide the knife in someone else’s bag | pretend the knife is part of a hip replacement | store the knife in an airport locker | take a train instead | threaten security with the knife | The airport security becomes suspicious of Carl, so they take him in custody and he ends up missing the festival. |
| 13 | Daniel is the ring bearer at his sister’s wedding. 10 minutes into the ceremony, he reaches into his pocket but the ring isn’t there. He realizes he left it on the seat next to him in the taxi. | You are the ring bearer at your sister’s wedding. 10 minutes into the ceremony, you reach into your pocket but the ring isn’t there. You realize you left it on the seat next to you in the taxi. | blame it on the maid of honor | borrow their mom’s wedding ring | call the taxi company | run away | steal someone else’s ring | tell his sister as soon as possible | When his sister finds out her ring is missing, she is disappointed and decides to cancel the wedding. |
| 14 | Andy’s mother just died and without her, his father is unable to take care of himself. Andy can’t afford to put his father in a retirement home and his apartment has barely enough room for one person. | Your mother just died and, without her, your father is unable to take care of himself. You can’t afford to put him in a retirement home and your apartment has barely enough room for one person. | abandon his father | buy his father a tent | check in on his father daily | get another job | move into his father’s apartment | pitch his situation to a reality TV network | After Andy does this, his father’s mental and physical health quickly deteriorates. |
| 15 | Ahmed is the captain of a 50-passenger cruise ship. Four days into their trip they get caught in a storm and the engine breaks under the pressure. They are now stranded at sea. A rescue boat is on its way but might not arrive for a week. Their food supplies are quickly dwindling and Ahmed isn’t sure what to instruct the passengers and crew. | You are the captain of a 50-passenger cruise ship. Four days into your trip, you get caught in a storm and the engine breaks under the pressure. You are now stranded at sea. A rescue boat is on its way but might not arrive for a week. The food supplies are quickly dwindling and you aren’t sure what to instruct the passengers and crew. | organize a fishing crew | ration their food | request supplies to be airdropped | sell all the food to the highest bidder | serve engine fuel to the passengers | throw disobedient passengers off the boat | After Ahmed tries this, several passengers suffer injuries from heatstroke and dehydration. |
| 16 | Eva and Todd are on a mountain climbing trip in Arizona. In the middle of their longest day Todd slips and gets his arm trapped in a crevice. Eva tries calling 911 but she has no service. The sun is beating down on them and Eva is worried that she might be getting dehydrated as she unsuccessfully tries to get his arm out of the rock. | You and your friend Todd are on a mountain climbing trip in Arizona. In the middle of your longest day Todd slips and gets his arm trapped in a crevice. You try calling 911 but you have no service. The sun is beating down on you and you’re worried that you might be getting dehydrated as you unsuccessfully try to get Todd’s arm out of the rock. | get her arm stuck in solidarity | give Todd her water | hike until she finds cell service | leave Todd to save herself | saw Todd’s arm off | yell at Todd for getting stuck | After Eva does this, Todd dies. |
| 17 | Jeff works the night shift at a 24/7 convenience store. Every day at 7am the owner of the store, Jeff’s boss, takes over for the morning shift. Today, Jeff has an important court appointment at 8am but his boss hasn’t shown up yet and isn’t returning his calls. | You work the night shift at a 24/7 convenience store. Every day at 7am the owner of the store, your boss, takes over for the morning shift. Today, you have an important court appointment at 8am but your boss hasn’t shown up yet and isn’t returning his calls. | ask a friend to take over | call his lawyer | leave the store unattended | lock up the store | stage a break-in at the store | tell his lawyer that he’s sick | After the owner returns, she fires him. |
| 18 | Shania is competing in a 100-mile dogsledding race in Minnesota. She hopes to come in first and use the prize money to pay for her college tuition. Halfway through the race she finds one of her competitors lying unconscious next to his crashed sled. | You are competing in a 100-mile dogsledding race in Minnesota. You hope to come in first and use the prize money to pay for your college tuition. Halfway through the race you find one of your competitors lying unconscious next to their crashed sled. | call for help | carry her competitor on her sled | finish the race | hide him in the snow | leave her dogs with him | steal his dogs for extra speed | When the competition judges find out what happened, they cancel the race and no one wins the prize money. |
Table 2
| group | Context_1 | Context_2 | Context_3 | Context_4 | Context_5 | Context_6 | Context_7 | Context_8 | Context_9 | Context_10 | Context_11 | Context_12 | Context_13 | Context_14 | Context_15 | Context_16 | Context_17 | Context_18 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| group_1 | ask family/friends | call a taxi | put his own money in | renew her membership | tell the teacher the truth | start a fire (not for signal) | go without him | go talk to the family | call 911 | ignore her | call 911 | tell security the truth | call the taxi company | have his father move in with him | ration food | leave and search for help | call his boss | call for help |
| group_2 | fundraise | call a tow truck/AAA | tell his manager the truth | use the front desk’s phone | call her mom | call for help | skip the concert | talk to the teen | pull over | give her a full ride | break the windows | throw the knife away | tell the bride/groom | move in with his father | prioritize who gets food | just leave him | call a coworker | keep going/ignore the situation |
| group_3 | seek government/charity assistance | hitchhike | lie about missing money/fudge numbers | borrow someone else’s phone | redo assignment before class | keep hiking | go home to find ticket | yell at/confront the teen | pass the car | take her part of the way | check if door is locked | attempt to mail the knife | borrow someone’s ring | find a new apartment | fish | stay with him | close the store | take competitor on her sled |
| group_4 | take out a loan | call family/friends for help | ask his coworkers about missing money | leave the gym | copy someone else’s homework | stay put/make shelter | give him her ticket | ignore the situation | take a detour | drive her for a price | notify mall staff | try to check the knife | stall the wedding | ask relatives for help | keep passengers calm | search for cell service/call 911 | wait for his boss | stop and help |
| group_5 | don’t get treatment | miss/cancel trip | pool money with coworkers | sneak in | turn nothing in/do nothing | kill/eat his friends | try to buy another ticket | call 911 | honk at the driver | refuse/say no | wait by the car | just hope they don’t find the knife | buy a new ring | get a new job | kill/force people off the boat | search for water | stay at the store and miss his appointment | help/call then keep going |
| group_6 | get another job | walk/run to the airport | ignore the missing money/do nothing | wait for her friend | skip class | search/hunt for food | help him sneak in | watch closely | slow down/avoid driver | lie about where he’s going | do nothing | postpone his flight | lie | seek state help | radio/signal for help | cut off his arm | just leave the store | stay until help arrives |
| group_7 | sell possessions | reschedule his flight | quit his job | ask to be let in for free | make up an excuse | find their way out/locate themselves | explain to venue/security | make a scene | ram into car/force off the road | call her a cab | search for owner | call a friend to pick it up | tell the officiant | check in on/take care of his father himself | lie to passengers about situation | keep trying to get his arm free | call the court and explain/reschedule | finish the race then tell someone |
| group_8 | steal medicine | call his insurance | check camera footage | get a day/guest pass | go home to get assignment | tell truth/ask friends for help | be rude to/attack Ted | yell to the family/get their attention | follow driver | give her money | contact animal control | rent an airport locker | fake a illness/faint | kill his father | steal the food for himself | signal for help | ask family/friend to cover for him | steal their dogs |
| group_9 | steal money | steal a car | recount money | go back to work (and don’t tell friend) | ask for an extension | flare/fire/signal for rescue crew | steal a ticket | take photos/record the teen | flash lights | yell at/curse at/scold her | shame the owner | leave it in his car | make a fake ring | seek charity help | turn to cannibalism | cover him for shade | quit his job | get help of other racers |
| group_10 | ask for a discount from pharmacy/company | call the airline | blame someone else | leave a message for her friend | cry | abandon his friends | stay in parking lot | attack the teenager | flag down a police officer | tell her to call a car herself | set off the alarm | just get out of line/leave | search for the ring | hire a caretaker | ask for advice | kill him | leave a voicemail explaining to his boss | go get help |
| group_11 | buy medicine anyway | ask a stranger for help | work the overtime himself | explain to her friend later | fake being sick | retrace their steps/find way back | call someone to bring the ticket | report to an authority nearby | make note of license number | pretend to be deaf | open the window | hide the knife | tell someone else | abandon/leave his father alone | explain the situation to passengers | yell for help | text his boss | check on him |
| group_12 | find cheaper alternative medication | take a bus | let staff work overtime | go back to office to call her friend | steal a classmate’s assignment | ration their food | check for digital receipt/ticket | note characteristics of the teen | do nothing | call the police | yell for help | go home | pretend he has the ring | steal money | pretend nothing is wrong | give up/accept their deaths | leave a note without locking the store | run over the competitor |
| group_13 | get new insurance | fix his car | search for the missing money | ask gym staff to contact her friend | get a note from her mom | do nothing | wait for him to get his ticket | leave | motion for driver to pull over | let her borrow his phone | ask other people nearby to help | pretend he didn’t know about it | run away | talk to his father | let the passengers vote | save their urine for drinking | call police to check on his boss | kill the competitor |
| group_14 | ask insurance provider to pay for it | call emergency services | steal money to make up for it | just go home | turn it in late | find a water source | also go through belongings | take a video | ask for more information | pick the lock | leave the knife at security | search for the taxi | crowdsource money | take the food for himself | wait until dark to do anything | call his boss’ family/friends | report at next opportunity | |
| group_15 | do nothing | overcharge remaining customers | find a payphone | split up | ask others nearby for help | call a friend | comfort the dog | turn the knife in | panic/cry | get a loan | fix the engine | |||||||
| group_16 | ditch his car | just get lunch | pull up next to and inspect driver | threaten someone with the knife | use his own ring | ask neighbors/friends for help | keep passengers entertained | |||||||||||
| group_17 | put it in someone else’s bag | do nothing | put him in a homeless shelter | give up his own food | ||||||||||||||
| group_18 | put him in a retirement home | |||||||||||||||||
| group_0 | other | other | other | other | other | other | other | other | other | other | other | other | other | other | other | other | other | other |